Papers with layer-wise analysis
Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Language confusion is a critical challenge for large language models, especially for English-centric models. |
| Approach: | They propose to combine behavioral benchmarking with neuron-level analysis to study language confusion. |
| Outcome: | The proposed approach matches multilingual alignment in confusion reduction for many languages and yields cleaner, higher-quality outputs. |
Layer-Aware Task Arithmetic: Disentangling Task-Specific and Instruction-Following Knowledge (2025.findings-emnlp)
Copied to clipboard
Yan-Lun Chen, Yi-Ru Wei, Chia-Yi Hsu, Chia-Mu Yu, Chun-Ying Huang, Ying-Dar Lin, Yu-Sung Wu, Wei-Bin Lee
| Challenge: | Large language models (LLMs) demonstrate strong task-specific capabilities through fine-tuning, but merging multiple fine- tuned models often leads to degraded performance due to overlapping instruction-following components. |
| Approach: | They propose a layer-wise approach that assigns layer-specific weights to task vectors based on their alignment with instruction-following or task-specific components. |
| Outcome: | The proposed approach outperforms existing methods in learning and forgetting tasks while preserving overall model utility. |
An Empirical Study on the Transferability of Transformer Modules in Parameter-efficient Fine-tuning (2022.emnlp-main)
Copied to clipboard
| Challenge: | Parameter-efficient fine-tuning is a computationally expensive process . introducing new parameters to an already-large model can be considered a drawback. |
| Approach: | They investigate the capability of different transformer modules in transferring knowledge from a pre-trained model to a downstream task. |
| Outcome: | The proposed methods show that each transformer module is a winning ticket . they show that with only 0.003% updateable parameters, they can show acceptable performance on target tasks. |
CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmark (2026.findings-acl)
Copied to clipboard
Daniil Gurgurov, Yusser Al Ghussin, Tanja Baeumel, Cheng-Ting Chou, Patrick Schramowski, Marius Mosbach, Josef Van Genabith, Simon Ostermann
| Challenge: | Understanding and controlling behavior of large language models (LLMs) is an important topic in multilingual NLP. |
| Approach: | They propose a lightweight parallel-question benchmark for evaluating language-forcing behavior in large language models across 32 languages. |
| Outcome: | The proposed benchmark measures language steering in 32 languages across 32 languages. |
Rendering Data Unlearnable by Exploiting LLM Alignment Mechanisms (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) are increasingly trained on massive, heterogeneous text corpora, raising serious concerns about the unauthorised use of proprietary or personal data during model training. |
| Approach: | They propose a data-level defence that renders text unlearnable to LLMs by injecting carefully designed alignment-triggering disclaimers into the models' alignment mechanisms. |
| Outcome: | The proposed approach exploits the models’ alignment mechanisms to prevent effective learning. |